In recent years, a system architecture so called thin client system has become widespread, which performs most processes including applications on a server apparatus side and transmits a desktop screen image and a result of execution to a client apparatus side so that only required minimum processes can be performed on the client side used by a user.
In the thin client system, a result of processing by the server apparatus and image data such as a desktop screen image stored in a frame buffer are transmitted to the client apparatus through a network in real time. By transmitting image data in real time, the client apparatus is capable of displaying the image data (data in a frame buffer) created by the server apparatus immediately.
In the thin client system, although the server apparatus serves as an image transmitting apparatus for transmitting image data and the client apparatus serves as an image receiving apparatus for receiving the image data, there may be a delay generated in data communication with the client apparatus depending on a condition of the network through which the transmission is performed.
Thus, there is suggested a transmitting method which reduces an influence of a communication delay between a server and a client. As one of conventional methods, there is known a method which enables avoiding a transmission delay by transmitting a sending packet from a transmitting apparatus for transmitting image data to a receiving apparatus for receiving the image data and receiving by the transmitting apparatus a reply of a reception response including a transmission time of the sending packet from the receiving apparatus, and increasing and decreasing a number of packets, when transmitting image data, according to a relationship between a predetermined threshold value and a response time calculated from the transmission time of the sending packet.
Moreover, as another conventional method, there is known a method which enables transmitting image data without waiting a response by transmitting image data of a transmitting target by a transmitting side apparatus to a receiving side apparatus via a network after redundant-encoding the image data according to an encoding rate, which is set based on a packet loss rate expected on the network, and decoding the original image data in the receiving side apparatus using the received encoded data, which has been redundant-encoded.
In the thin client system, it is necessary to quickly detect image data stored in a frame buffer on the server apparatus side and cause the client apparatus to receive the detected image data while preventing the image data from being affected as much as possible.
However, the image data created by the server apparatus relates to a user operation generated in the client apparatus. Thus, a relationship between the user operation generated in the client apparatus and the image data displayed in the client apparatus may be unnatural depending on a timing of transmitting by the server apparatus the image data, which indicates a result of execution in response to the user operation.
When transmitting image data from a server apparatus to a client apparatus, there may be used a method of transmitting with a fixed frame rate and a method of transmitting after waiting for a response from a receiving side. However, a transmission delay is generated even when these transmitting methods are used.
FIG. 10 is an illustration for explaining the method of transmitting with a fixed frame rate.
The left part of FIG. 10 illustrates a server apparatus side of a thin client system, and the right part illustrates a client apparatus side.
When a mouse operation event occurs by a user, the client apparatus transmits each of mouse operations op1, op2, op3, op4, . . . to the server apparatus in an order of occurrence. The server apparatus transmits frames fr1, fr2, fr3, fr4, . . . , which are whole of a part of frame data of a desktop screen image to the client apparatus at a fixed transmission interval. The desktop screen image is updated according to an execution result of an application based on a user operation.
If the transmission interval of the server apparatus is long (if a frame rate is low), there is a problem in that a smooth screen updating cannot be performed because a reception interval becomes long in the client apparatus. For example, in the client apparatus, when the mouse operations op1 and op2 are performed by a user, a desktop screen (frame fr1) which does not correspond to the mouse operation op2 is displayed, or the frames fr2, fr3 and fr4 are displayed at a time when a considerable time has passed after the mouse operations op3 and op4 were done. Thus, there may be a case where a user has an uncomfortable feel that the desktop screen is being updated despite that the user has ended the operation.
If the transmission interval is short (if a frame rate is high), it is possible that a transmission is performed by exceeding a transmittable band of a network through which the server apparatus performs a transmission, and, thereby, reception of the image data (frame fr) is delayed in the client apparatus side, which may result in a delay in updating the desktop screen.
FIG. 11 is an illustration for explaining a delay in a case of the method of transmitting after waiting for a response from the receiving side.
The left part of FIG. 11 illustrates the server apparatus side of the thin client system, and the right part illustrates the client side apparatus. Similar to the case illustrated in FIG. 10, the client apparatus transmits a mouse operation op to the server apparatus in an order of occurrence, and the server apparatus transmits the image data (frame fr) of the desktop screen to the client apparatus after checking a reception response from the client apparatus.
If, for example, an RTT (Round Trip Time) delay is large in a network, the reception response is also delayed in the server apparatus, and, thus, there may be a case where a transmission of a frame fr is skipped depending on a timing of the reception response. For example, if the reception response of the frame fr1 from the client apparatus is delayed, the server apparatus transmits data stored in a frame buffer at a time of reception of the reception response, and, thus, the frame fr2 is skipped depending on a timing and the screen is updated in an order of the frames fr1 and fr3 in the client apparatus, which may prevent a smooth screen updating.
If the image data transmitted by the server apparatus is a desktop screen image or a screen image representing an execution result of an application, the server apparatus is required to transmit image data corresponding to an operation event as early as possible because the screen is updated according to the operation events, which occur in the client apparatus. However, in a case where a plurality of operation events occurs in a short time on the client apparatus side, if the screen to be updated is transmitted to the client apparatus, the transmission time becomes long depending on a network condition, which results in a generation of a transmission delay.
Here, in the condition in which a plurality of operation events occur in a short time, it is possible to regard that a user is performing a series of operations. The screen indicating a final processing result is more important for a user than a screen indicating a processing result in the middle of a series of user operations. Moreover, a user does not have an uncomfortable feel in display updating when a long time is spent on screen updating including the middle of a series of user operations rather than when a final screen is displayed even if the display updating in the middle of a series of user operations is omitted. Further, it is hard to have a feel that a response is delayed if a time until a time of display updating for a final screen is not long.